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Abstract 



In depth from defocus (DFD), when images are captured with different camera parameters, a relative 
magnification is induced between them. Image warping is a simpler solution to account for magnifica- 
tion than seemingly more accurate optical approaches. This work is an investigation into the effects of 
magnification on the accuracy of DFD. We comment on issues regarding scaling effect on relative blur 
computation. We statistically analyze accountability of scale factor, commenting on the bias and effi- 
ciency of the estimator that does not consider scale. We also discuss the effect of interpolation errors on 

C/2 ' blur estimation in a warping based solution to handle magnification and carry out experimental analysis 

^, , to comment on the blur estimation accuracy. 



> ! 1 Introduction 

CN I In DFD, relative defocus blur between images is exploited as a cue for depth [1]. However, accurately 

estimating blur is a difficult problem. After early works [TJ [5] , there are now many methods for estimating 
blur addressing major issues such as noise [3] , space- variance [31 15] and the blur model [4] . 

Cn . When images are captured with different camera settings to induce a relative blur, a relative magnifi- 

cation is also induced between them causing a shift between the corresponding pixels in the images. Since 
conventional DFD does not factor in such a shift, the blur estimation can be erroneous in presence of magni- 
fication (scaling) . Approaches to handle scaling can be classified as optical 6^ or warping based [71 |S] . Nayar 
and Watanabe [6] perform an optical correction by using an additional aperture at an analytically calculated 

^^ I position. But this adds to hardware and cost. Ghita ct al. 7 consider magnification correction in an active 

^ ■ DFD scheme by interpolation that exploits an active illumination pattern. An interpolation based approach 

proposed by Darell and Wohn [8] for shape from focus computes the warping by observing the motion of a 
special pattern. Scaling in DFD depends only on camera parameters and since it is required that camera 
parameters be known, the scale factor is available apriori. Warping based methods may be preferable due to 
simplicity but estimation could be compromised due to interpolation effects [6]. However, a formal analysis 
of the magnification effect in DFD is totally lacking in literature. 

In this paper, we formally analyze the magnification effect in DFD theoretically and experimentally. 
Section 2 points out important implications of magnification on the relative blur computation. Section 3 
provides a statistical analysis of scaling on blur estimation. Since the analysis implicitly assumes an image 
warping solution, section 4 discusses effects of interpolation errors. Section 5 provides experimental analysis 
regarding blur estimation accuracy and we conclude in section 6. We limit ourselves to space-invariant blur 
since our goal is not to propose a new method but to analyze magnification as an important consideration 
in DFD. 



2 Magnification in DFD 

For a thin lens model, the blur radius R and depth D can be related as 



— '^7-i 



(1) 



where r is the aperture radius, V is the distance between the lens and the image plane, F is the focal length. 
Varying r and F keeping V constant will cause relative blurring without inducing magnification. However, 
since _F is a function of the physical parameters of lens, changing F means physically changing the lens. As 
mentioned in [6 , the sensitivity of blur due to change in r is low. Hence a better way to induce relative 
blur is to vary V . However, this introduces a relative magnification between the two images. The scaling 
factor can be derived as s = ^, where Vi and V2 are two instances of V while capturing the two images 
(Fig 1). Although the above scaling factor is defined for a thin lens, our intention is to note that scale is a 




Figure 1: Magnification and blurring with variation in camera parameters 

function of known camera parameters. Hence, even in compound lenses, the scale factor can be known but 
with more involved calculations. Thus the amount of magnification can be known apriori. We now discuss 
two important implications of relative scaling on estimating blur. 



2.1 Order of scaling and blurring 

Due to magnification, two types of transformations are involved that relate the two images viz. scaling and 
blurring. The order of the transformations is vital when estimating the blur. Typically, the image formation 
is expressed as, 

g{x)^h{x)*f{x)+j^ix) (2) 

In frequency domain, we have 

G{lu) = H{lu)F{lu) + N{uj) (3) 

In DFD, the Gaussian point spread function (PSF) is popular as a blur model because the effects of blurring, 
diffraction and sampling can be expressed as a Gaussian in view of the central limit theorem [T] . The standard 
deviation a of the Gaussian PSF referred to as 'blur parameter' is related to R in equation (1) as cr = pR 

where p is a constant. Thus H{lj) — exp 2~. 

Considering the order of blurring followed by scaling, equation (3) becomes, 



G(u) ^-H iPj F {Pj + N{u) 



(4) 



Hence, H{—) = cxp^~275~. This effectively means that scaling of the image / is followed by blurring with 
blur parameter — . However referring to Figure 1 and its related discussion, we see that the shift in the pixels 
is due to scaling s and the blur around these corresponding pixel is given by equation (1). According to the 
above thin lens model, these shifted points are spread by a blur having parameter a rather than — . Thus 
we conclude that the order of scaling followed by blurring is more valid than vice-versa. Thus the image 
formation when considering scaling should be 

G{u) = -H{u)f(-\+N{uj) (5) 

s \ s J 

2.2 Effect of scaling on relative blur computation 

Generally, given two blurred images gi and 52 a model that is often used [5] is 

52(2;) = K(x) * gi{x) + ri{x) (6) 

where * denotes convolution. This can be derived as follows 

G^{uj) = H,{oj)F{oj) 1^1,2 (7) 

Gi[uj) Hi(lu) 

^G2(w) = Hr{uj)Gi{uj) 

In spatial domain adding noise this gives equation (6). The blur parameter of the relative blur hr{x) (or 
Hr{u))) is ycrl — af where cti and (72 are the blur parameters of Hi{uj) and H2{lo). This can be deduced 
from the Gaussian PSF of the blur and the fact that Hr{u}) = WTuA' Estimating blur from equation (6) 
actually means solving for -^al — af. This relationship along with 0-2 — ctai +/3, can be used to solve for ci 
or (72. Here a and /? are dependent on camera parameters as a = ^j^ and /3 = pr iVi (^ — -^ — T' ~^ V') 
in a general case of varying all the camera parameters ^2| . Considering scale in the frequency domain version 
of equation (6) results in a simple extension of (5) as 

G2(w) = -Hr{uj)Gi (-) + iV(w) (8) 

s \ s / 

Apparently, this should resulting in the estimation of a/ctI — af, the blur parameter of Hr{uj). However 
such an extension of (5) is incorrect as shown below. Actually, the scale consideration should be in the basic 
image formation i.e. equation (7), 

G,H = -H,{uj)f(-] z-1,2 (9) 



Si \ s 



G2M = ^-^^Gif^. 



S2 



S2 



(10) 



Here, si and S2 are the scaling factors when the focused image / is transformed into gi and (72 respectively. 
We note that since we do not have the focused image, we also do not have si and S2- However, we only need 
the relative scale s = — which is known apriori from the camera parameters. We now write the modified 
version of equation (6) as, 

32(2;) = hr{x) * g2{sx) + ri{x) (11) 

Here, the correct relative blur parameter of hj. is -^/ctI — s^a\ rather than \J (T2 ~ o'l- Thus when accounting 
for scaling one must consider this modification in the relative blur expression to solve for u\ or 172 • 



3 Analysis of the scaling effect 

In this section we answer the foUowing question. 'Given two images with relative scahng and blurring, how 
important is scale consideration for blur estimation ?' We show that the Least-Squares (LS) estimator that 
does not account for scaling is both biased and inefficient. For simplicity, we assume that one of the two 
images is the focused image. 

3.1 Bias in blur estimation 

From the discussion in the previous section, the true image formation model can be written as 

g{x) = h{x) * f{sx) + rj{x) or g — Fgh + 77 (12) 

where Fs is a block Toeplitz image matrix corresponding to the focused image f{sx), g is the observed image, 
his a. blur vector and 77 is a zero mean additive white Gaussian noise (AWGN) vector. A least-squares solution 

of the blur estimate h will be 

h^{FjF,)-^Fjg (13) 

The bias in this estimate is then 

h^ = E{h)~h = E{{FjFs)-^Fjg)~h^{FjFs)-^FjFsh-h = {) (14) 

However, if scaling is not taken into account during estimation, then we obtain the estimator 

h={F^Fy'F^g_ (15) 

Since in actuality, E{g) ~ Fgh, this results in a non zero bias if one does not account for the scale factor s. 

b^E{h)-h = E{{F^Fy^F^g)^h^{F^F)-'^F'^Fsh-h^O (16) 

The above bias can be explained as follows. The matrix F consists of entries from the focused image /(x). 
The matrix Fg consists of entries from /(sx), the scaled version of the focused image. To account for scale, 
one must convert /(x) to /(sx). Doing this will ideally make the bias to be {F^ Fs)~^ F]" Fgh ~ h = 0. Not 
accounting for scale will mean that we are using the focused image /(x) for estimating the blur when the 
actual focused image is / (sx), the scaled version of the /(x), thus inducing a bias. 

3.2 Efficiency of the estimator 

Here we comment on the efficiency of the estimator that does and does not consider scaling. The efficiency is 
in the sense of the Cramer- Rao lower bound (CRLB). We do not provide the details due to space constraints. 
Considering the true image formation (equation (12)), the CRLB for the blur estimate h can be shown to 
be 

Covih) > a2(FjF,)-i (17) 

where Cov{h) is the covariance matrix oi h and cr^ is the noise variance. The LS estimator h — {F^ Fs)^^F^ g 

is an efficient estimator that meets the CRLB. The estimator that does not use the scaled image is /i = 
{F F)~^F g. As shown above, this estimator incurs a bias b. Suppose a new estimator which is formed 

by subtracting the bias b from h — (F^F)^^F^g i.e. knew = {F^ F)~^ F^ g - b. This new estimator will be 
unbiased for obvious reasons. However, covariance matrix of this estimator turns out to be, 

CovChnew) > <yl{F^F)-^ (18) 



Since this is not equal to the CRLB, it is an inefficient estimator. Ideahy, to achieve an unbiased and efficient 
estimate of blur parameter a, we must utilize the scale factor s to transform the observed focused image 
f{x) to its scaled version f{sx) and then compute a by solving equation (12) in the least-squares sense. 
However, in practice, the transformation of f{x) to f{sx) involves image interpolation. In the next section 
we comment on the effects of interpolation errors on the blur estimate. 

4 Interpolation errors 

Till now, we implicitly assumed ideal image warping to alleviate the magnification effect, i.e. the images 
are exactly aligned after warping. However, due to interpolation this will not be so. The difference between 
the ideal scaled image fis{x) and interpolated version fs{x) is what we call 'interpolation noise'. 



fs{x) = fis{x) +rii{x) 



(19) 



The samples of rii{x) are correlated as explained next. In Figure 2 pixels -P^s and Af^s belong to the reference 
image and warped image, respectively. Considering bilinear interpolation as an example, the pixels P2,Pb 
contribute to both Mi and M2 ■ Such common contributions during interpolation induce correlation in rji {x) . 
Also, from the histograms in Figure 3, we can empirically deduce that the pdf of ri{x) is heavy tailed and may 
not be well approximated as Gaussian. Thus interpolation errors induces correlation and non-Gaussianity. 
From equations (12), (19) and considering ideal scaling in equation (12) 



^{x) = h{x) * fisix) + ri{x) = h{x) * fs{x) - h{x) * 77,; (x) + ri{x) 



(20) 



where h{x)*7]i{x) is the correlated, non-Gaussian component of the distribution. The LS estimator considered 
in the previous section is efficient for AWGN jlO , and not when the distribution possesses non-Gaussianity 
and uncorrelatedness [11] |T2]. However a class of robust M-estimators are shown to be asymptotically 
efficient even under such pathologies [HI [13] . We have explored the performance of some M-estimators for 
blur estimation. In the next section, we carry out experiments and analyze the blur estimation accuracy. 
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Figure 3: Histograms of rii{x) for three images 



5 Results 



We experimentally analyzed blur estimation accuracy versus variation in scale, blur and noise. We include 
two discontinuity adaptive functions and an absolute difference as robust estimators in our experiments. 



Discontinuity adaptive functions can serve as robust estimators due to the analogy between discontinuities 
and outliers [14]. Thus the estimators that we experiment with are the least squares estimator (LSE), 
absolute difference estimator (ABS) and discontinuity adaptive functions (DAl and DA2) 



LSE: argmin_E(2;) 

ABS: argmin_E(2;) 

DAI: argmin_E(x) 

DA2: argmin_E(2;) 



argmin((7i(a;) 
argmin \g\{x) 

(7 

argmin (1 



argmin 1 



hr{x) *g2{sx)Y 
hr{x) * g2{sx)\ 
exp^'^^^'^^^''''''^^*^^'*'^^^^) 

1 

1 + (51(2;) -hr{x) *g2{sx)f 



(21) 



Variation in scale factor: We vary the scale factor from 0.7 to 0.95 keeping the blur constant. The blur 
estimates were computed for various constants values of ui and 172 • Figure 4(a) shows the results for ui — 0.7 
and (72 = 1-2. We observe that there is no proper behavior of the estimated blur with scale variation. This is 
because the interpolation errors do not depend on the magnitude of the scale but rather on the shifts of in- 
dividual pixels. We observe that ABS and DA estimators are very accurate as compared to the Least squares. 

Variation in blur: This experiment involves variation of blur with a constant scale factor. In figure 
4(b) we show the results for scale factor of 0.9 with ci constant at 0.7 and (T2 varying from 0.7 to 1.5. It 
is quite clear that generally the inaccuracy increases with blurring. However, again the inaccuracy is quite 
negligible for ABS, DAI and DA2 estimators. 

Variation in noise: We experimented with noise variances from 1 to 25 with constant scale and blur. 
Fig 4(c) shows results for s — 0.9, ci = 1, (T2 = 1-5. For the LSE, the error is fairly constant but large. For 
the ABS estimator, the error increases with noise but it is quite small. The DA estimators incur negligible 
error. 
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Figure 4: Blur estimation results (a) Scale variation (b) Blur variation (c) Noise variation 

Thus, estimation accuracy in warping based solution depends lot on the estimator. The LS estimator is 
inaccurate as interpolation errors cause violation of the underlying assumptions concerning the pdf. However, 
robust estimators are very accurate despite interpolation effects. 



6 Conclusion 

This work analyzed the inherent magnification effect in DFD. Important issues such as the order of scaling - 
blurring and the effect of scaling on relative blur computation were discussed. We then carried out statistical 



analysis and concluded that the estimator that does not handle scaling is both biased and inefficient. Since an 
image warping solution was inherently assumed, we scrutinized the effect of interpolation on blur estimation 
accuracy. We conclude that blur estimation using robust estimators performs very well. 
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